A Default First Order Family Weight Determination Procedure for WPDV Models

Weighted Probability Distribution Voting (WPDV) is a newly designed machine learning algorithm, for which research is currently aimed at the determination of good weighting schemes. This paper describes a simple yet effective weight determination procedure, which leads to models that can produce competitive results for a number of NLP classification tasks. 1 T h e W P D V a l g o r i t h m Weighted Probability Distribution Voting (WPDV) is a supervised learning approach to classification. A case which is to be classified is represented as a feature-value pair set: Fcase -{{fl : V l } , . . . , { f n :Vn}} An estimation of the probabilities of the various classes for the case in question is then based on the classes observed with similar feature-value pair sets in the training data. To be exact, the probability of class C for Fcase is estimated as a weighted sum over all possible subsets Fsub of Fcase: w /req(CJF b) P(C) = N(C) /req(F b) FsubCFcase with the frequencies (freq) measured on the training data, and N(C) a normalizing factor such that ~ / 5 ( C ) = 1. In principle, the weight factors WF,~,~ can be assigned per individual subset. For the time being, however, they are assigned for groups of subsets. First of all, it is possible to restrict the subsets that are taken into account in the model, using the size of the subset (e.g. Fsub contains at most 4 elements) and/or its frequency (e.g. Fsub occurs at least twice in the training material). Subsets which do not fulfil the chosen criteria are not used. For the subsets that are used, weight factors are not assigned per individual subset either, but rather per "family", where a family consists of those subsets which contain the same combination of feature types (i.e. the same f/). The two components of a WPDV model, distributions and weights, are determined separately. In this paper, I will use the term training set for the data on which the distributions are based and tuning set for the data on the basis of which the weights are selected. Whether these two sets should be disjunct or can coincide is one of the subjects under investigation. 2 F a m i l y w e i g h t s The various family weighting schemes can be classified according to the type of use they make of the tuning data. Here, I use a very rough classification, into weighting scheme orders. With 0 th o r d e r we igh t s , no information whatsoever is used about the data in tuning set. Examples of such rudimentary weighting schemes are the use of a weight of k! for all subsets containing k elements, as has been used e.g. for wordclass tagger combination (van Halteren et al., To appear), or even a uniform weight for all subsets. With 1 st o r d e r we igh t s , information is used about the individual feature types, i.e. WF,~b = IT WIt { i l ( f i :v i}eF, ub} First order weights ignore any possible interaction between two or more feature types, but